Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach

نویسندگان

  • Todd Kulesza
  • Margaret M. Burnett
  • Simone Stumpf
  • Weng-Keen Wong
  • Shubhomoy Das
  • Alex Groce
  • Amber Shinsel
  • Forrest Bice
  • Kevin McIntosh
چکیده

Intelligent assistants are handling increasingly critical tasks, but until now, end users have had no way to systematically assess where their assistants make mistakes. For some intelligent assistants, this is a serious problem: if the assistant is doing work that is important, such as assisting with qualitative research or monitoring an elderly parent’s safety, the user may pay a high cost for unnoticed mistakes. This paper addresses the problem with WYSIWYT/ML (What You See Is What You Test for Machine Learning), a human/computer partnership that enables end users to systematically test intelligent assistants. Our empirical evaluation shows that WYSIWYT/ML helped end users find assistants’ mistakes significantly more effectively than ad hoc testing. Not only did it allow users to assess an assistant’s work on an average of 117 predictions in only 10 minutes, it also scaled to a much larger data set, assessing an assistant’s work on 623 out of 1,448 predictions using only the users’ original 10 minutes’ testing effort.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

I-36: Preimplantation Genetic Diagnosis - Where Have We Been and Where Are We Going

Preimplantation genetic diagnosis (PGD) is now considered routine in IVF laboratories with micromanipulation capability and access to genetic diagnostic services. The past two decades have witnessed a dramatic increase in the use of PGD, the number of cycles performed, and the indications for which PGD has been used. This increase has been mirrored by a slow, but steady, increase in the range o...

متن کامل

Classifying and Detecting Plan-Based Misconceptions for Robust Plan Recognition

My Ph.D. dissertation (Calistri 1990) extends traditional methods of plan recognition to handle situations in which agents have flawed plans.1 This extension involves solving two problems: determining what sorts of mistakes people make when they reason about plans and figuring out how to recognize these mistakes when they occur. I have developed a complete classification of plan-based misconcep...

متن کامل

Abbas Kiarostami, Family Film and the Techno-cultural Processes of Transcultural Viewing

In this paper Abbas Kiarostami's films for children are discussed from the perspective of a cognitive studies approach. The crux of the argument is that the visual elements of film are essentially metonymic. Where is My Friend’s Home? has a quest-script instantiated by means of four components drawn on Brown and Babbington. Transcultural viewing is enabled by techno-cu...

متن کامل

An Intelligent System’s Approach for Revitalization of Brown Fields using only Production Rate Data

State-of-the-art data analysis in production allows engineers to characterize reservoirs using production data. This saves companies large sums that should otherwise be spend on well testing and reservoir simulation and modeling. There are two shortcomings with today’s production data analysis: It needs bottom-hole or well-head pressure data in addition to data for rating reservoirs’ characteri...

متن کامل

Intelligent Virtual Assistant's Impact on Technical Proficiency within Virtual Teams

Information-systemsdevelopmentcontinuestobeadifficultprocess,particularlyforvirtualteams thatdonothavetheluxuryofmeetingface-to-face.Theresearchliteratureonthistopicreinforces thispoint: thegreaterpartofdatabasesystemsdevelopmentprojectsends in failure.Theuseof virtualteamstocompleteprojectsfurthercompoundsthesefailures.However,recentdevelo...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2011